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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  & 
Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 
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•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 

•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
Department  of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 


We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  symposium. 


James  B.  Greene  Jr. 

Rear  Admiral,  U.S.  Navy  (Ret.) 


Keith  F.  Snider,  PhD 
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Panel  18.  Root  Cause  Trends  in  Program  Cost 
Growth 


Thursday,  May  17,  2012 

11:15  a.m.  - 
12:45  p.m. 

Chair:  Mr.  Gary  R.  Bliss,  Director,  Performance  Assessments  and  Root  Cause 
Analyses,  Office  of  the  Assistant  Secretary  of  Defense  for  Acquisition 

Lessons  Learned  From  Seven  Root  Cause  Analyses 

Charles  Nemfakos,  RAND  Corporation 

Root  Causes  Associated  With  Program  Execution 

David  L.  McNicol,  Institute  for  Defense  Analyses 

Forensic  Studies  to  Understand  Project  Performance 

James  N.  Ortiz,  National  Aeronautics  and  Space  Administration 

Gary  R.  Bliss — Mr.  Bliss  is  the  director  of  Performance  Assessments  and  Root  Cause  Analyses 
(PARCA)  in  the  Office  of  the  Assistant  Secretary  of  Defense  for  Acquisition.  PARCA  carries  out 
performance  assessments  of  Major  Defense  Acquisitions  Programs  (MDAPs)  and  conducts  root 
cause  analyses  for  MDAPs  with  Nunn-McCurdy  breach  status  or  when  requested  by  senior 
Department  of  Defense  (DoD)  officials. 

Mr.  Bliss  previously  held  the  position  of  deputy  director  of  Enterprise  Information  and  the  Office  of 
the  Secretary  of  Defense  (OSD)  Studies  in  the  Office  of  the  Under  Secretary  of  Defense  for 
Acquisition,  Technology,  and  Logistics  (USD[AT&L]).  His  responsibilities  included  oversight  of  the  five 
OSD-funded  Federally  Funded  Research  and  Development  Centers,  the  OSD’s  university  research 
program,  as  well  as  review  and  development  of  innovations  to  overhaul  the  AT&L  enterprise 
management  systems. 

Earlier  in  his  career,  Mr.  Bliss  served  13  years  as  the  director  of  the  Office  of  the  Director, 

Program  Analysis,  &  Evaluation  Weapon  System  Cost  Analysis  Division  (WSCAD).  WSCAD’s  10  staff 
members  constitute  one  of  the  two  offices  dedicated  to  OSD  Cost  Analysis  Improvement  Group 
(CAIG)  functions,  and  is  responsible  for  the  preparation  of  independent  development  and 
procurement  cost  estimates  for  major  systems  that  range  from  munitions  (e.g.,  tactical  missiles) 
through  platforms  (e.g.,  helicopters,  submarines,  fighter  aircraft,  tanks,  etc.).  As  such,  Mr.  Bliss  has 
been  a  key  player  in  the  DoD’s  most  important  system  decisions  by  the  Services,  the  OSD,  and  the 
Congress. 

Generally  recognized  in  both  industry  and  government  as  a  leading  authority  on  the  economics  of 
defense  procurement,  Mr.  Bliss  has  an  established  track  record  in  institutional  reform/reengineering. 
He  is  often  asked  to  speak  to  varied  audiences  on  these  topics,  including 

•  management  information  system  governance  and  reengineering, 

•  manufacturing  enterprise  reengineering,  and 

•  acquisition  institutional  reform. 

Mr.  Bliss  has  a  BA  in  mathematics  and  economics  (highest  honors  in  economics)  from  The 
College  of  William  and  Mary. 
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James  N.  Ortiz — Dr.  Ortiz  joined  NASA  from  the  U.S.  Air  Force  where  he  served  as  flight  test 
engineer  for  Electronic  Warfare  systems  on  high  performance  aircraft.  He  was  the  lead  flight  test 
engineer  on  the  F-15  E  fighter  aircraft  leading  to  its  initial  operational  capability  and  deployment  to 
Desert  Storm.  During  his  career  at  NASA,  Dr.  Ortiz  has  held  several  assignments  in  the  International 
Space  Station  (ISS);  as  senior  NASA  research  exchange  engineer  with  the  Air  Force  Research 
Laboratory;  as  chief  of  the  Advanced  Projects  Office  for  Missions  Operations;  and  manager  of  the 
JSC  Systems  management  office.  Dr.  Ortiz  served  as  the  focal  point  for  design  certification  of  the 
space  shuttle  fleet  for  return  to  flight  after  the  Columbia  accident  and  he  later  served  as  deputy 
manager  for  integrated  avionics  and  software  for  the  Orion  project  developing  the  next  generation  of 
human  space  vehicle.  He  was  selected  as  director  of  IPAO  in  September  2009.  Dr.  Ortiz  holds  a  PhD 
in  electrical  engineering  from  the  University  of  Houston,  a  Master  of  Engineering  degree  in  electrical 
engineering  from  the  University  of  Florida,  and  a  Master  of  Science  degree  in  systems  engineering 
from  the  Naval  Postgraduate  School  in  Monterey,  CA.  Dr.  Ortiz  is  a  member  of  the  Senior  Executive 
Service.  [James.n.ortiz@nasa.gov] 

Abstract 

The  purpose  of  this  paper  is  to  present  a  summary  of  studies  being  conducted  by  the 
Independent  Program  Assessment  Office  (IPAO)  to  understand  overall  trends  in  project 
performance  so  the  results  of  the  studies  provide  information  on  any  needed  improvements 
to  agency  policies,  training,  or  capabilities.  The  IPAO  is  responsible  for  the  independent 
review  and  assessment  of  NASA  programs  and  projects  at  designated  stages  in  the  lifecycle 
with  the  results  of  these  assessments  provided  to  the  project  management  governing  boards 
and  used  in  support  of  approval  decisions  at  key  decision  points  in  the  development  and 
operations  lifecycle.  The  forensics  studies  described  in  this  paper  look  across  the  findings 
and  recommendations  reported  by  the  individual  review  teams  looking  for  systemic  trends  in 
project  performance  that  may  provide  indications  of  areas  where  projects  are  doing  well  and 
areas  where  projects  are  finding  difficulties.  The  paper  describes  the  methodology 
implemented,  the  status  of  the  study,  some  of  the  preliminary  results  and  lessons  learned, 
and  a  description  of  the  way  forward. 

Introduction 

NASA  programs  and  projects  undergo  a  series  of  comprehensive  independent 
assessments  as  part  of  the  approval  process  at  designated  stages  in  the  development  and 
operations  lifecycle.  These  independent  lifecycle  reviews  are  required  by  NASA  policies 
(NASA,  2007a,  2007b)  and  are  conducted  by  independent  review  teams,  also  known  as 
Standing  Review  Boards  (SRBs),  and  performed  in  accordance  with  processes  and 
methodologies  designated  by  the  Independent  Program  Assessment  Office  (IPAO;  NASA, 
2009).  SRBs  are  composed  of  a  multidisciplinary  group  of  experts  who  are  independent 
from  the  management  chain  of  the  project  or  programs  and  who  perform  assessments 
covering  the  full  spectrum  of  project  content  including  technical,  cost,  schedule,  and  risk. 
SRBs  assess  the  project  or  program  readiness  to  transition  to  the  next  stage  in  the  lifecycle 
using  criteria  that  is  specified  in  NASA  policy,  and  the  results  of  each  assessment  include 
positive  findings  (strengths)  and  negative  findings  categorized  in  descending  level  of 
criticality  as  issues,  concerns,  and  observations.  Approximately  thirty  SRB  reviews  are 
conducted  and  reported  on  a  typical  year. 

The  impetus  for  the  forensics  study  was  the  recognition  that  while  SRB  assessments 
are  reported  for  each  individual  review,  more  information  is  contained  in  the  aggregate  of  all 
reports  and  this  information  could  provide  a  picture  of  the  systemic  performance  of  the 
agency’s  projects  and  programs.  This  information,  in  turn,  could  help  shed  light  into  the 
effectiveness  of  policy  initiatives  directed  to  improve  project  performance,  or  the  need  to 
improve  methodologies,  training,  or  core  competencies. 
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Methodology 

As  mentioned  in  the  previous  section,  the  forensics  study  was  undertaken  to  “mine” 
individual  reports  produced  by  the  SRBs  assessing  NASA  programs  and  projects,  looking 
for  systemic  trends  in  performance.  A  decision  was  made  during  the  planning  phase  of  the 
study  to  present  results  at  the  aggregate  level  across  all  programs  and  projects  with  no 
information  that  could  be  tied  to  a  specific  program  or  project.  The  intent  was  to  attempt  to 
identify  trends  in  project  performance  across  the  agency’s  portfolio  and  not  to  highlight  the 
shortcoming  of  any  specific  program  or  project.  As  the  study  got  underway  in  2010,  it  was 
also  decided  that  this  study  should  be  done  in  an  evolutionary  fashion,  in  other  words,  that 
the  study  was  going  to  be  conducted  in  phases  with  improved  analytical  capabilities 
incorporated  as  the  results  of  the  previous  phase  were  analyzed  and  lessons  learned 
incorporated.  The  first  phase  of  the  study  consisted  of  the  examination  of  findings  and 
recommendations  contained  in  the  SRB  reports  from  2008-2010.  The  first  order  or  sorting 
consisted  of  analyzing  positive  or  negative  findings  contained  in  these  reports  against  the 
following  six  criteria  elements  contained  in  NASA  polices  and  listed  in  Table  1. 


Table  1 .  Assessment  Criteria  Used  for  First  Phase  of  the  Study 


Criteria  Element 

Description 

Goals 

Alignment  with  and  contributing  to  Agency  needs,  goals, 
and  objectives,  and  the  adequacy  of  requirements  flow- 
down  from  those 

Technical 

Adequacy  of  technical  approach,  as  defined  by  NPR 
7123.1  entrance  and  success  criteria 

Budget 

Adequacy  of  estimated  costs  (total  and  by  fiscal  year), 
including  independent  cost  analyses  (ICAs)  and 
independent  cost  estimates  (ICEs),  against  approved 
budget  resources 

Schedule 

Adequacy  of  schedule 

Resources 

Adequacy/availability  of  resources  other  than  budget 

Risk 

Adequacy  of  risk  management  approach  and  risk 
identification/mitigation 

Management 

Adequacy  of  management  approach 

The  IPAO  analysts  developed  a  simple  spreadsheet  to  count  the  frequencies  of  each 
type  of  finding  for  strengths  and  weaknesses  using  the  data  from  54  reviews  over  a  three- 
year  period  (2008-2010).  The  frequency  information  was  used  to  develop  frequency  tables 
and  graphs  to  show  the  distribution  of  positive  and  negative  findings  across  the  criteria 
elements.  As  the  results  of  the  frequency  distributions  were  analyzed,  the  data  was  then 
sliced  in  different  ways  to  provide  insights  into  groupings.  Accordingly,  the  data  was  grouped 
by  year,  by  performing  field  center,  by  broad  area  of  work  (science  robotics  missions  or 
human  spaceflight),  and  by  lifecycle  milestone. 

The  results  of  the  analyses  were  reported  broadly  but  provided  only  a  limited  level  of 
additional  insight.  The  study  team  realized  the  need  to  penetrate  to  a  level  deeper  in  the 
criteria  elements  in  order  to  attempt  to  understand  high-level  root  cause.  Understanding  of 
root  cause  could  then  provide  a  closer  correlation  and  insight  into  any  needed  areas  of 
improvement,  which  remain  the  primary  goal  of  the  study. 

The  study  team  began  the  second  stage  of  the  analysis  in  June  201 1  with  the 
formulation  of  an  updated  categorization  criteria  looking  for  first-order  root  cause.  The  effort 
began  by  incorporating  an  updated  set  of  criteria  that  the  agency  had  adopted  which 
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included  more  detailed  information  on  maturity  expectations  programs  and  projects  needed 
to  demonstrate  to  meet  the  criteria  at  each  stage  of  the  lifecycle.  The  study  team 
decomposed  each  criteria  element  into  approximately  six  sub-levels.  Each  sub-level  was 
given  five  statements  to  help  guide  the  classification  of  positive  or  negative  findings  into  the 
more  detailed  sub-levels  or  buckets.  An  example  of  the  decomposition  for  the  Technical 
criteria  element  is  provided  in  Table  2. 


Table  2.  Sub-Level  Statements  for  Decomposition  of  Technical  Element  Criteria 


Technical  Criteria  Element 

Adequacy  of  technical  approach,  as  defined  by  NPR 
7123.1  entrance  and  success  criteria. 

Causes  of  Strength 

Mission  architecture  and  designs  close  with 
Program/project  requirements 

Operations  concepts  close  with  mission  designs  and 
achieve  mission  needs 

Demonstrated  design  maturity  is  achieved  for  lifecycle 
stage 

Technology  needs  achieve  proper  level  of  maturity  to 
support  downstream  development  and  integration 

Test,  verification,  and  integration  results  are  consistent 
with  plans  and  support  schedule  and  cost  commitments 

Effective  integration  processes  in  place 

Causes  of  Issues/Concerns 

Mission  architecture  and  designs  do  not  close  with 
program/project  requirements 

Operations  concepts  do  not  close  with  mission  designs 
and  achieve  mission  needs 

Demonstrated  design  maturity  is  not  achieved  for 
lifecycle  stage 

Technology  needs  do  not  achieve  proper  level  of 
maturity  to  support  downstream  development  and 
integration 

Test,  verification,  and  integration  results  are  not 
consistent  with  plans  and  support  schedule  and  cost 
commitments 

Lack  of  effective  integration  processes 

After  the  more  detailed  criteria  was  formulated  and  agreed  upon,  it  was  then  piloted 
using  a  subset  of  SRB  reports.  Six  SRB  reports  from  the  2010  timeframe  were  used  for  the 
pilots.  After  the  findings  and  recommendations  were  extracted  from  the  selected  reports  and 
classified  using  the  new  detailed  criteria,  there  were  positive  indications  that  enhanced 
levels  of  insight  were  provided  using  the  new  detailed  criteria.  The  team  also  noted  that  the 
data  was  growing  significantly,  potentially  exceeding  the  practicality  of  storing  the  data  in 
spreadsheets  and  possibly  requiring  the  use  of  a  relational  database.  At  this  point,  due  to 
staffing  limitations  in  the  IPAO,  the  study  was  put  in  hold. 

Results 

A  forensic  analysis  of  SRB  findings  was  conducted  by  the  IPAO  using  data  from  54 
reviews  over  a  three-year  period  (2008-2010).  The  ultimate  purpose  of  the  forensic  analysis 
was  to  identify  trends  and/or  systemic  issues  in  terms  of  which  programs  and  projects  at 
NASA  are  doing  well  and  not  doing  well.  It  was  thought  that  with  the  results  of  the  analysis, 
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it  might  be  possible  to  affect  policies  and  procedures  to  better  ensure  success  across  the 
Agency.  During  the  initial  phase  of  the  study,  a  subjective  analysis  of  the  data  was  executed 
based  on  strength/weakness,  frequency,  and  consistency  of  findings  per  each  assessment 
criterion  in  NASA  program  management  policy.  All  reviews  conducted  by  the  IPAO  in  2008 
to  2010  were  targeted.  SRB  Final  Reports  and  Management  Briefings  were  the  sources  for 
data  (including  18  2008  reports/briefings,  22  2009  reports/briefings,  and  14  2010 
reports/briefings).  The  data  was  sliced  by  year,  center,  directorate,  and  selected  milestones. 
The  results  of  the  first  phase  of  the  study  are  summarized  in  Figure  1 . 


Figure  1.  Summary  Results  From  the  First  Phase  of  the  Forensics  Study 


This  first  phase  of  the  analysis  found  that  NASA  appears  to  have  offsetting  strengths 
and  weaknesses  with  respect  to  the  Technical  and  Management  criteria,  and  that  more  work 
needs  to  be  done  in  terms  of  identifying  cross-cutting  themes  given  the  limited  insight 
provided  by  the  classification  approach  using  during  this  phase  (to  be  addressed  as  part  of 
the  second  phase  of  the  study).  The  study  also  concluded  that  schedule  preparation, 
analysis,  and  management  may  be  the  areas  that  present  the  best  opportunity  for 
improvement,  and  that  risk  management  appears  to  be  the  area  where  NASA  most 
consistently  excels. 


The  following  recommendations  resulted  from  the  first  phase  of  the  analysis: 


•  Emphasize  the  programmatics.  Strive  for  better  balance  between  emphasis 
on  technical  excellence  and  cost  and  schedule  performance. 

•  Enable  realistic  planning. 

•  Provide  sufficient  budget/resources  to  programs  and  projects  to  better  enable 
success. 

•  Encourage  the  use  of  good  schedule  practices. 
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•  Ensure  a  more  disciplined  flow-down  of  Level  1  Requirements. 

Reports  of  the  first  phase  of  the  forensics  analysis  were  presented  to  the  Program 
and  Project  Management  Board  (PPMB)  led  by  the  Office  of  the  Chief  Engineer,  and  at  the 
NASA  Program  Management  Challenge  Conference  along  with  forward  plans  to  extend  the 
analysis. 

Current  Status 

As  discussed  in  the  previous  section,  the  forward  plan  for  the  second  phase  of  the 
study  included  updating  the  criteria  for  classification  in  the  most  recent  version  of  NASA 
program  management  policy  and  further  detailing  the  criteria  by  formulating  sub-categories 
looking  for  first-order  root  cause.  This  effort  got  underway  and  completed  the  update  to  the 
NASA  policy  criteria  and  the  development  of  subcategories  under  the  updated  criteria. 
Additionally,  a  pilot  was  performed  using  a  limited  number  (six)  of  2010  SRB  reports  that 
demonstrated  that  the  new  sub-categorization  was  providing  increasing  levels  of  insight.  At 
this  point,  in  July  201 1 ,  the  effort  was  halted  due  to  workforce  constraints  in  the  IPAO. 

The  study  was  restarted  in  late  March  2012,  and  will  continue  from  the  work 
completed  during  the  second  phase  pilot  and  will  include  analyzing  of  the  findings  for  the 
SRB  2010  and  2011  reports  and  categorizing  these  findings  using  the  more  detailed  criteria 
looking  at  first-order  root  cause  of  areas  of  strengths  and  weaknesses. 

Conclusion 

This  paper  summarized  studies  being  conducted  by  the  IPAO  to  understand  overall 
trends  in  project  performance  to  provide  information  on  any  needed  improvements  to 
agency  policies,  training,  or  capabilities.  The  paper  described  the  methodologies 
implemented,  the  status  of  the  study,  some  of  the  preliminary  results  and  lessons  learned, 
and  a  description  of  the  way  forward. 
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Purpose  of  Forensic  Analysis 

•  The  ultimate  purpose  of  this  analysis  is  to  identify 
trends  and/or  systemic  issues  in  terms  of  what  NASA 
is  doing  well  and  not  doing  so  well  in  managing 
Program  and  projects 

•  With  the  results  of  this  analysis,  it  may  be  possible  to 
affect  policies  and  procedures  that  better  ensure 
success  across  the  Agency 

“In  the  past,  NASA  has  had  difficulty  meeting  cost,  schedule, 
and  performance  objectives  for  many  of  its  projects.  The  need 
to  effectively  manage  projects  will  gain  even  more  importance 
as  NASA  seeks  to  manage  its  wide-ranging  portfolio  in  an 
increasingly  constrained  fiscal  environment.”  -  GAO, 

Assessments  of  Selected  Large-Scale  Projects,  Feb  2010 
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Background 

•  NASA  programs  and  projects  undergo  a  series  of  comprehensive 
independent  assessments  as  part  of  the  approval  process 

•  These  independent  lifecycle  reviews  are  required  by  NASA  policies  and 
conducted  by  independent  review  teams,  known  as  Standing  Review 
Boards  (SRBs) 

•  The  impetus  for  the  forensics  study  was  the  recognition  that  while  SRB 
assessments  are  reported  for  each  individual  review,  more  information  is 
contained  in  the  aggregate  of  all  reports  and  this  information  could 
provide  a  picture  of  the  systemic  performance  of  the  agency's  projects  and 
programs  ("data  mine  the  SRB  reports") 

•  This  information  in  turn,  could  help  shed  light  into  the  effectiveness  of 
policy  initiatives  directed  to  improve  project  performance,  or  the  need  to 
improve  methodologies,  training,  or  core  competencies 

•  The  Independent  Program  Assessment  Office  (IPAO)  is  responsible  for  the 
independent  review  and  assessment  of  NASA  programs  and  projects  at 
designated  stages  in  the  lifecycle  to  support  approval  decisions  at  key 
decision  points  in  the  lifecycle 


Methodology 


Study  being  performed  in  phases 
(evolutionary) 

Results  are  presented  at  the 
aggregate  level 

1st  phase  was  based  SRB  findings 
against  agency  criteria  from  54 
reviews  over  a  three  year  period 
(2008-2010). 

Frequency  information  was  used 
to  develop  frequency  tables  and 
graphs  to  show  the  distribution  of 
positive  and  negative  findings 
across  the  criteria  elements 


Criteria  Element 

Description 

Goals 

Alignment  with  and  contributing  to  Agency  needs,  goals,  and 
objectives,  and  the  adequacy  of  requirements  flow-down  from 
those. 

Technical 

Adequacy  of  technical  approach,  as  defined  by  NPR  7123.1 
entrance  and  success  criteria. 

Budget: 

Adequacy  of  estimated  costs  (total  and  by  fiscal  year), 
including  Independent  Cost  Analyses  (ICAs)  and  Independent 
Cost  Estimates  (ICEs),  against  approved  budget  resources 

Schedule: 

Adequacy  of  schedule 

Resources 

Adequacy/availability  of  resources  other  than  budget 

Risk 

Adequacy  of  risk  management  approach  and  risk 
identification/mitigation 

Management 

Adequacy  of  management  approach. 

NASA  Criteria 
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Analysis  results  (phase  1) 


Management 


Risk 


Resources 


Schedule 


Budget 


Technical 


Goals 


140 


■  Issues  &  Concerns 

■  Concerns 

■  Issues 

B  Strengths 
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Summary  results  (phase  1) 

•  NASA  appears  to  have  offsetting  strengths  and  weaknesses 
with  respect  to  the  Technical  and  Management  criteria 

-  Communication  and  Integration  may  be  areas  for  improvement 

-  Next  generation  of  data  analysis  methodology  should  produce  more 
actionable  results  (phase  2) 

•  Schedule  preparation,  analyses,  management  may  be  the  area 
that  presents  the  best  opportunity  for  improvement 

-  IPAO  has  seen  moderate  but  steady  improvement  with  respect  to  quality  of 
schedules. 

•  Risk  Management  appears  to  be  the  area  where  NASA  most 
consistently  excels 

-  35%  received  at  least  one  strength;  9%  received  at  least  one  issue;  17% 
received  either  an  issue  or  concern 
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Recommendations  (phase  1) 

•  Emphasize  the  "programmatics" 

—  Strive  for  better  balance  between  emphasis  on  technical 
excellence  and  cost  and  schedule  performance 

•  Enable  realistic  planning 

•  Provide  sufficient  budget/resources  to  programs  and 
projects  to  better  enable  success 

•  Encourage  the  use  of  good  schedule  practices 
(training) 

•  Ensure  a  more  disciplined  flow-down  of  Level  1 
Requirements 
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Current  Status 

•  2nd  phase  underway 

•  Emphasis  is  on  a  deeper  level  of  understanding 

-  Looking  for  1st  order  root  cause 

•  Updated  classification  criteria  for: 

-  Most  recent  version  of  NASA  program  management  policy 

-  Further  detailing  the  criteria  by  formulating  sub-categories  (See  an 
example  of  the  following  page) 

•  A  pilot  performed  using  a  limited  number  (six)  of  2010  SRB 
reports  demonstrated  the  new  sub-categorization  was 
providing  increasing  levels  of  insight 

•  Analyses  of  2010  and  2011  reports  underway 
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Expanded  criteria  for  phase  2 


Technical  Criteria 

Element 

Adequacy  of  technical  approach,  as  defined  by  NPR  7123.1 
entrance  and  success  criteria. 

Causes  of  Strength 

Mission  architecture  and  designs  close  with  Program/project 
requirements 

Operations  concepts  close  with  mission  designs  and  achieve 
mission  needs 

Demonstrated  design  maturity  is  achieved  for  lifecycle  stage 

Technology  needs  achieve  proper  level  of  maturity  to  support 
downstream  development  and  integration. 

Test,  verification  and  integration  results  are  consistent  with 
plans  and  support  schedule  and  cost  commitments 

Effective  integration  processes  in  place 

Causes  of  Issues/Concems 

Mission  architecture  and  designs  do  not  close  with 
Program/project  requirements 

Operations  concepts  do  not  close  with  mission  designs  and 
achieve  mission  needs 

Demonstrated  design  maturity  is  not  achieved  for  lifecycle 
stage 

Technology  needs  do  not  achieve  proper  level  of  maturity  to 
support  downstream  development  and  integration. 

Test,  verification  and  integration  results  are  not  consistent 
with  plans  and  support  schedule  and  cost  commitments 

Lack  of  effective  integration  processes 
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Summary 

•  This  briefing  described  studies  being  performed  by  the  IPAO 
to  understand  overall  trends  in  project  performance  to 
provide  information  on  any  needed  improvements  to  agency 
policies,  training,  or  capabilities 

•  The  accompanying  paper  describes  in  more  detail  the 
methodologies  implemented,  the  status  of  the  study,  some  of 
preliminary  results  and  lessons  learned,  and  a  description  of 
the  way  forward 
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Definitions  of  Findings 

per  SRB  Handbook 

•  Strength:  A  strength  is  a  finding  of  the  SRB  that  describes  a 
feature  of  the  P/p  that  in  the  judgment  of  the  SRB  is  better 
than  expected  at  a  particular  stage  of  the  life-cycle. 

•  Issue:  A  finding  by  the  SRB;  SRB  issues  are  documented  and 
briefed  to  the  P/p  and  the  management  councils;  issues 
typically  drive  the  SRB's  success  criteria  assessment  and 
ultimate  determination  of  the  SRB  rating  for  each  review. 

•  Concern:  A  finding  identified  by  the  SRB;  SRB  concerns  are 
typically  documented  and  briefed  to  the  P/p,  but  not 
specifically  addressed  with  the  management  councils  (unless 
asked). 
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